
Department of Homeland Security 

Office of Inspector General 



Review of Management Agreements 
Developed for DHS 1 Primary Data Center 




OIG-10-56 



February 2010 



Office of Inspector General 



U.S. Department of Homeland Security 

Washington, DC 20528 



FEB 1 8 2010 




% Homeland 
I Security 



MEMORANDUM FOR: 



Richard Spires 
Chief. Information Officer 



FROM: 



Richard L. Skinner 
Inspector General 




SUBJECT: 



Final Letter Report: Review of Management Agreements 
Developed for DHS ' Primary Data Center (OIG- 10-56) 



We audited the efficiency of the Department of Homeland Security's (DHS) management 
agreements to establish a primary data center at the Stennis Space Center in Mississippi. 
The department has agreements with the U.S. Navy to establish and maintain the DHS 
primary data center. We determined that the associated interagency agreements for this 
effort are missing key elements. Additionally, the department's technical representatives 
have not reviewed invoices associated with $160 million in expenditures. 

The audit included a review of DHS policies and procedures, interagency agreements, the 
memorandum of agreement, and prior audit reports. We are recommending that DHS: 

• Update its interagency agreements to include (a) the location of the building; 

(b) DHS' percentage of space, power, and utilities; and (c) the methodology for 

the allocation of costs. 
« Review invoices and supporting documentation for fund transfers to ensure that 

funds were appropriately used. 

Background 

In April 2009, we reported on DHS' efforts to improve its disaster recovery planning for 
information systems. 1 We identified technical and managerial concerns that might hinder 
the ability of DHS' primary data center to perform mission-essential functions. As a 
result of that audit, we initiated a review to determine whether the management 
agreements among the various stakeholders at the Stennis Space Center assist or hinder 
DHS' efforts to establish a primary data center. 



1 DHS ' Progress in Disaster Recovery Planning for Information Systems, OIG-09-60, April 2009. 
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We reviewed actions DHS had taken to select the location of its primary data center. In 
2004, DHS staff visited the Stennis Space Center in Mississippi several times to survey 
proposed buildings that could be used for the data center. In November 2004, the DHS 
Chief Information Officer committed $5 million to establish data center services at the 
Stennis Space Center. However, the buildings that DHS staff had surveyed were not 
available, and the U.S. Navy provided DHS another building at the Stennis Space Center. 
The new building was a former munitions manufacturing facility that needed extensive 
renovations. This building is on land permitted to the U.S. Army and operated by the 
U.S. Navy. DHS is one of several tenants in the U.S. Navy building. 

We reviewed the interagency agreements between DHS and the U.S. Navy. We also 
reviewed the process DHS uses to verify that goods and services received for the primary 
data center are approved before payments are made. 

Interagency Agreements Do Not Contain Key Elements 

Since 2005, the department has obligated more than $160 million in interagency 
agreements with the U.S. Navy to convert a former munitions manufacturing facility at 
the Stennis Space Center in Mississippi into a data center. However, the interagency 
agreements between DHS and the U.S. Navy do not contain key elements essential to 
ensure that the facility can continue to operate as DHS' primary data center. 

According to Management Directive 125-02, Interagency Agreements, 

"Each Interagency Agreement identifies the supplies and services to be provided 
and indicates the fee charged by the Servicing Agency. The description of 
supplies or services must be specific, definite, and clear in order to support a 
binding agreement that will be recorded as an obligation. Further, the Interagency 
Agreement must establish a ceiling amount limiting the Requesting Agency's 
financial obligation." 

Key elements that are missing from these agreements include DHS' allotted floor space, 
the amount of power and utilities provided, and DHS' percentage of shared costs. 
Without these items in the interagency agreements, DHS cannot be assured that it can 
continue to house its primary data center in the current location, or that DHS is 
reimbursing the U.S. Navy only for expenses directly related to the primary data center. 

In November 2004, DHS committed to placing its primary data center at the Stennis 
Space Center without establishing an agreement with the U.S. Navy about the specific 
building that DHS would occupy. After DHS committed $5 million to use the Stennis 
Space Center, DHS was provided another building that it had not previously reviewed. 
According to the U.S. Navy, DHS spent $64.1 million to renovate the replacement 
building. 

In September 2008, DHS and the U.S. Navy signed a formal memorandum of agreement 
for the operation and maintenance of this renovated facility. This document and the 
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original interagency agreements specify what services the U.S. Navy will provide. 
However, these documents do not specify the amount of floor space that DHS will 
control or DHS' percentage of shared operating costs. 

Our review of DHS' payments indicates that the department is paying more than its fair 
share of costs for this facility. The DHS primary data center is located in a multitenant 
building. This building is operated by the U.S. Navy on land that has been permitted to 
the U.S. Army. The U.S. Army is charging the U.S. Navy $549,332 for the use of 
250,973 square feet of office space on 64 acres of land at the Stennis Space Center. 
According to the U.S. Navy, DHS is occupying approximately 81%, or 61,206 square 
feet, of the floor space in one of its shared buildings. However, the U.S. Navy is 
charging DHS $444,959, or approximately 81% of the U.S. Navy's $549,332 total cost, 
instead of charging DHS based on the percentage of space DHS is using. We discussed 
this matter with DHS officials, who agreed that DHS may be paying the U.S. Navy too 
much for this space. 

Without documented agreements identifying the fees to be charged, as well as the 
specific supplies and services to be provided, DHS cannot be assured that it is operating 
its primary data center as cost-effectively as possible. 

DHS Is Not Reviewing Invoices for Its Primary Data Center 

DHS technical representatives have not reviewed invoices and other documents that 
detail the expenditures associated with the $160 million that DHS has paid the U.S. Navy 
for its primary data center since January 2005 . Without reviews of supporting 
documentation, senior DHS management cannot be assured that department funds are 
being used to acquire the goods and services outlined in the interagency agreements in a 
cost-effective manner and for authorized purposes. 

The U.S. Navy is using the Intra-Governmental Payment and Collection (IP AC) System 
to obtain DHS funds designated for DHS' primary data center. 2 However, the U.S. Navy 
did not provide supporting documentation or invoices for these transfers before receiving 
the funds. In addition, DHS technical representatives did not request supporting 
documentation for the transactions and did not notify the DHS contracting officers, fund 
transfer officials, or budget officers that supporting documentation was not provided. 

According to an U.S. Navy official, the IP AC System was used to obtain DHS funds 
made available through the interagency agreements for capital improvements. For 
example, as discussed above, DHS funds were used for renovating a munitions 
manufacturing facility and for rent. Additionally, according to a U.S. Navy official, DHS 
funds were used for rent; salaries; and the purchase of nonexpendable property such as air 
conditioning units (chillers), uninterruptible power supplies, fuel tanks, electrical power 
supply elements, and five of six emergency generators. However, DHS was not provided 
the invoices or documentation supporting charges for these goods and services. 



2 The IP AC System's primary purpose is to provide a standardized interagency fund transfer mechanism for 
Federal Program Agencies. 
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According to Management Directive 125-02, Interagency Agreements, 

"The Program Manager is ultimately responsible for ensuring that program funds 
are spent in a manner that results in planned outcomes (i.e., that all Interagency 
Agreements contain clearly defined requirements, and performance measures and 
monitoring plans to the maximum extent practicable). Further, DHS Program 
Offices must ensure that adequate documentation is maintained with respect to the 
results achieved under each Interagency Agreement." 

According to DHS Directive 125-02-001, Instruction Guide on Interagency Agreements, 
the following activities are to occur before the close-out phase of an interagency 
agreement: 

"Prior to close-out, reconciliation of the funds and acceptance of performance 
must occur (Requesting Agency Program Manager in conjunction with the 
appropriate Servicing Agency Point of Contact)." 

Without supporting documentation, DHS cannot be assured that IPAC transfers to the 
U.S. Navy related to the DHS primary data center were for authorized goods or services 
and were appropriately used for capital improvements, rent, salaries, and nonexpendable 
property. 

Recommendations 

We recommend that the DHS Chief Information Officer (CIO) direct DHS technical 
representatives and contracting officers to: 

Recommendation #1 : Update interagency agreements between the U.S. Navy and DHS 
to include (a) the location of DHS' data center building; (b) DHS' percentage of space, 
power, and utilities; and (c) the methodology for determining DHS' allocation of costs 
for space at the Stennis Space Center. 

Recommendation #2 : Obtain and review invoices and supporting documentation for 
U.S. Navy's IPAC transfers related to the DHS primary data center to ensure that funds 
were appropriately used for designated goods and services. 



Page 4 



Management Comments and Analysis 



We obtained written comments on a draft of this report from the DHS CIO. In the 
comments, the CIO indicates that the report contains several inaccuracies. Specifically, 
the CIO states that two specific interagency agreements contained power and floor space 
requirements. We disagree. Neither agreement specifies the amount of floor space or 
power that will be dedicated for DHS' use. Additionally, the other 29 agreements we 
reviewed did not contain the amount of floor space and power that would be dedicated to 
DHS' use. 

Further, the CIO indicated that our report appeared to confuse raised floor space with 
other types of space. We modified the text in this section of the report to emphasize our 
concern that DHS was paying 81% of the U.S. Navy's total costs, including costs for 
buildings not occupied by DHS. 

Finally, the CIO asserted that onsite surveys and visual inspections were sufficient 
actions to ensure that program funds were properly spent. However, it is our opinion that 
these actions alone did not provide sufficient oversight. These actions must be supported 
by verifying that adequate documentation exists for costs charged to DHS for 
construction, supplies, and personnel. 

The CIO also did not concur with our second recommendation. Specifically, the CIO 
noted that both DHS and the Office of Inspector General auditors sought invoices from 
the U.S. Navy without success. However, we disagree that DHS should have accepted 
this apparent lack of cooperation from the U.S. Navy. Staff from the DHS Office of the 
CIO should have notified senior DHS officials of this problem. When informed of the 
problem, DHS officials should have negotiated with the U.S. Navy and the General 
Services Administration for access to the relevant documents. 
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Appendix A 

Purpose, Scope, and Methodology 



This audit is a follow-up audit to our report on DHS' Progress in 
Disaster Recovery Planning for Information Systems (OIG-09-60). 
The previous report concerned DHS' disaster recovery planning 
and its two new data centers. This report focuses on the DHS 
Management Directorate's efforts to establish one of those data 
centers, DHS' primary data center. 

The objective of our audit was to determine whether the 
management/ownership agreements among the various 
stakeholders assist or hinder DHS' efforts to establish a primary 
data center. We reviewed DHS policies and procedures, 
interagency agreements, memorandum of agreement, statements of 
work, and prior audit reports. On-site inspections and interviews 
of key personnel performed for the previous report were also used 
to support this effort. 

We conducted fieldwork at DHS Management Directorate 
facilities and organizational elements in the Washington DC, 
metropolitan area. We also requested information from the 
U.S. Navy, the U.S. Army, and the National Aeronautics and 
Space Administration. While these organizations provided us with 
most of the data we requested, the U.S. Navy did not provide 
invoices related to work performed at DHS' primary data center. 
We conducted this audit between April and October 2009. 

We provided DHS staff with briefings and presentations 
concerning the results of fieldwork and the information 
summarized in this report. We conducted the audit according to 
generally accepted government auditing standards. Those 
standards require that we plan and perform the audit to obtain 
sufficient, appropriate evidence to provide a reasonable basis for 
our findings and conclusions based on our audit objectives. We 
believe that the evidence obtained provides a reasonable basis for 
our findings and conclusions based on our audit objectives. 

We appreciate the efforts of DHS management and staff to provide 
the information and access necessary to accomplish this audit. The 
principal Office of Inspector General points of contact for the audit 
are Frank Deffer, Assistant Inspector General for Information 
Technology Audits, (202) 254-4100, and Sharon Huiswoud, 
Director, Information Systems Division, (202) 254-5451. Major 
contributors to the audit are identified in Appendix C. 
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Appendix B 

Management Comments to the Draft Report 



OJJtm 0/ Inspector CSmmd 



I .S. Dcpurtincnl of llonicliiml StCWUj 

Washington. DC 31)528 



Homeland 
Security 



MEMORANDUM FOR: 



Richard L. Skinner 
Inspector General 




FROM: 



Richard Spires / 1^ 
Chief Informat ioiy O fine er 



SUBJECT: 



Final Letter Report: Review of Management Agreements 
Developed for DHS ' Primary Dalu < enter Response 



Please find our comments on the subject report. In addition to responding to the 
recommendations we have addressed several inaccuracies that were noied in the report. 

Page 2, fifth paragraph ("Key elements...") - DHS' requirements for power and floor 
space were specified in Interagency Agreements HSHQPA-05-X-OI066 and HSHQPA- 
05-X-01066, PQ0002. The current Memorandum of Agreement (MOA) with the 
Commander Naval Meteorology and Oceanography Command (CNMOCl specifies that 
annual budget for facility services will be proposed and mutually discussed. Interagency 
agreements for recurring facility costs prior and subsequent lo this MOA have specified 
services and costs. 

Pa»e 3. first I'ul! psiruyrapli ("Our review..,") ['he report appears i<> confuse raised 
floor space with other types of space. DHS uses a number of buildings and types of 
space within the former Mississippi Army Ammunition Plant (MSAAP). Raised floor 
space, as the premium facility commodity, is used by CNMOC as the determining metric 
tor costs cited in this paragraph. DHS currently has exclusive use of 81". i of the raised 
floor space within the MSSAP. 

Page 4, initial reference to Management Directive 125-02 - DHS personnel undertook 
repeated on-site surveys of the facility during the course construction, performing a 
thorough visual inspection of assets such as chillers, fuel tanks, power supply elements 
and emergency generators. Such inspection represents full attention to the MD 125-02 
requirement lo ensure program funds arc properlj spent. 

Recommendation #1 : t update interagency agreements between the U.S. Navy and DHS 
to include: (a) the location of DHS data center (building), (b) DHS' percentage of space. 
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power, and utilities, and (c) the methodology for determining Dl IS' allocation of costs 
for space at the Stennis Space Center. 

OCIO Concurs with caveat - Facility ownership is expected to transfer from CNMOC 
to the NASA Stennis Space Center (SSC) in April 201 0. DHS and SSC are presently 
working to establish an agreement for specific terms and conditions. Dl IS plans to 
incorporate this recommendation into the new agreement with SSC. 

RecoiimietitUitioit #2 : Obtain ami review invoices and supporting documentation for 
U.S. Navy 's IPAC transfers related to the DHS primary data center to ensure that funds 
were appropriately used for designated goods and services. 

OCIO Non-concurs - Dl IS has sought invoices from Navy without success. The DHS 
Office of Inspector Genera) also sought invoices without success. It is not a common 
practice for invoices to be provided in the kinds of i nlra-go v ernmental relationships such 
as that DHS has for the data center at SSC. Contracting functions related to facility 
services are performed by the Navy and the General Services Administration (GSA), thus 
access to invoices is controlled by CNMOC Contracting Officer Representatives and 
GSA Contracting Officers. These parlies have direct responsibility for vendor oversight 
and invoice review. Further, these parlies have treated invoice information as subject to 
privity. As DHS works with SSC to establish a new agreement, efforts will be made to 
impose a requirement for access to vendor invoices, but results eannot be assured. DHS 
has and will continue to mitigate this risk through direct on-site observation of rendered 
services and capabilities. 
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Appendix C 

Major Contributors to This Report 



Sharon Huiswoud, Director, Department of Homeland Security, 
Information Technology Audits 

Kevin Burke, Audit Manager, Department of Homeland Security, 
Information Technology Audits 

Domingo Alvarez, Senior Auditor, Department of Homeland 
Security, Information Technology Audits 

Matthew Worner, Senior Auditor, Department of Homeland 
Security, Information Technology Audits 

Philip Greene, Referencer 
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Appendix D 

Report Distribution 

Department of Homeland Security 

Secretary 

Deputy Secretary 

Chief of Staff for Policy 

Chief of Staff for Operations 

Deputy Chiefs of Staff 

General Counsel 

Executive Secretary 

Under Secretary for Management 

Director, GAO/OIG Liaison Office 

Assistant Secretary for Policy 

Assistant Secretary for Legislative Affairs 

Assistant Secretary for Public Affairs 

Office of Management, Audit Liaison 

Chief Information Officer, DHS 

Deputy Chief Information Officer, DHS 

Chief Information Security Officer, DHS 

Director, Compliance and Oversight 

Deputy Director, Compliance and Oversight 

DHS Chief Information Officer, Audit Liaison 

DHS Chief Information Security Officer, Audit Liaison 

Office of Management and Budget 

Chief, Homeland Security Branch DHS OIG Budget Examiner 
Congress 

Congressional Oversight and Appropriations Committees, as 
appropriate 
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ADDITIONAL INFORMATION AND COPIES 

To obtain additional copies of this report, please call the Office of Inspector General (OIG) at (202) 254-4100, 
fax your request to (202) 254-4305, or visit the OIG web site at www.dhs.gov/oig. 



OIG HOTLINE 

To report alleged fraud, waste, abuse or mismanagement, or any other kind of criminal or noncriminal 
misconduct relative to department programs or operations: 

• Call our Hotline at 1-800-323-8603; 

• Fax the complaint directly to us at (202) 254-4292; 

• Email us at DHSOIGHOTLINE@dhs.gov; or 

• Write to us at: 

DHS Office of Inspector General/MAIL STOP 2600, 
Attention: Office of Investigations - Hotline, 
245 Murray Drive, SW, Building 410, 
Washington, DC 20528. 



The OIG seeks to protect the identity of each writer and caller. 



